Software Installation

The Quantum VM installer consists of an OVA image which contains both the installation VM and the corresponding ActiveScale ISO. Download the appropriate OVA file to your laptop.

3.1.1 When You Can't Use the OVA

In case OVA can't be used (MacBook with Apple M1/M2, VM provided by 3rd party, ESXi host, ...) you can also use the TGZ archive.

Refer to 3.7 Setup Installer via TGZ on Your Own VM for instructions.

Note: If you use the TGZ archive, you will still need to run through the other sections listed below, as they remain applicable, regardless if you use OVA or the TGZ archive.

The VM Installer can be imported by double-clicking the downloaded file and importing it via the VirtualBox Wizard. All default values should suffice. Alternatively, the OVA can also be imported via the Command Line on Unix based host systems by executing:

To be able to communicate with the ActiveScale system, the host system needs to able to communicate in Private VLAN1 for X200 and P200 or Private VLAN2 for P100E3. To achieve this, the host system which will run the VM installer will need to be physically connected to the switch which also connects the nodes of the ActiveScale system.

The VM installer will connect to this network by using the bridged networking mode of VirtualBox. We recommend attaching an extra VNIC to the just-imported VM so communication between the host system and the VM is easier. An extra benefit of this is the fact that by default the VNIC of the VM is in NAT-networking mode which means it can access all systems which the hosts system can reach (including the Internet if applicable).

To attach an extra VNIC to the VM installer you can use either the UI of VirtualBox or run the following commands on the host system:

Note: eth666 in the above example is the physical NIC of the host systems which is connected to the appropriate Private VLAN of the ActiveScale system.

Note: This port should be located in PrivateVLAN1 for X200 and P200 and in PrivateVLAN2 for P100E3!

To do this via the VirtualBox UI, you need to edit the just-imported VM and add a new VNIC. Make sure to select the correct mode: BRIDGED and connect it to the physical NIC of the host system which is connected to the ActiveScale VLAN. In most, if not all cases, this will be the Ethernet interface of the laptop.

Leave the already present VNIC in NAT-mode. This makes life simpler to connect to the installer, and enables the installer to reach the Internet if the laptop itself has Internet access through e.g. WiFi.

Note: Internet access is not a requirement to perform the install, but it’s a nice to have to sync things like time over NTP.

Once the above network config is completed, it is time to start the VM. This can be done by using the UI of VirtualBox or run the following command on the host system:

This will start the Ubuntu VM. Please note that by default the newly added bridged networkadapter is not connected to the VM. Once the VM has booted, it is time to connect it to the VM by using the UI or running on the host system:

Note: If you opted to add an extra bridged VNIC, you can now reach the VM via SSH over the default VNIC in NAT mode.

Final step of the setup of the VM installer itself is to assign an IP in the Private VLAN of the ActiveScale system. Important to note here is that this IP adresss should be unique in the VLAN. This means that it should not be in use by any device in the VLAN and it cannot be assigned to any of the to-be-installed nodes. It should be a valid IP in the subnet of the Private VLAN.

By default for X200 and P200, this is Private VLAN1, VLANid 1000, 172.20.0.0/24. We recommend using: 172.20.0.240

By default for P100E3, this is Private VLAN2, VLANid 210, 172.16.210.0/24. We recommend using: 172.16.210.240

Note: These subnets/VLANids can differ depending of the customer’s requirements. But it’s important that for X200 and P200 the installer can reach Private VLAN1 and for P100E3 the installer can reach Private VLAN2

This IP can be set via the Ubuntu UI of the VM by navigating to Settings > Network and selecting the correct NIC. If you opted for the 2 VNIC solution, by default 1 NIC in the Ubuntu VM will have an IP in 10.0.2.0/24. Use the other NIC to set the IP in the Private VLAN. In most situations, the bridged NIC has the name ‘enp0s8’ in the Ubuntu VM.

Via commandline, the IP can be set in the VM by running the following command on the VM itself:

Or alter the config file under /etc/netplan/network-config.yaml as root to:

And the run the following command on the VM:

Note: By using the Netplan approach, the IP is persistent across reboots of the VM.

Because an ActiveScale system does not like major time jumps, we sync the date and time of the to-be installed ActiveScale nodes with the time defined on the VM Installer during installation. If Internet access is available to the VM installer, it can sync over NTP with a timeserver and no explicit steps should be needed.

Note: Always double check the date and time defined on the VM installer before continuing.

However if there’s no Internet available and/or you see the date/time in the VM installer being off as oposed to the actual UTC time, it’s a good idea to set the clock correctly via the VM Installer UI. We don’t need split-second precision, but we do want to have the correct hour and ofcourse date set, so the ActiveScale system has a nice starting point and does not need to jump time later on.

In case OVA can't be used (MacBook with Apple M1/M2, VM provided by 3rd party, ESXi host, ...) one can also use the TGZ archive.

This sections assumes the following:

• Sections 3.5 and 3.6 above were executed.

• You have downloaded the .tgz file and extracted it to the ~/quantum_installer on the VM.

To begin, you will want to install the following packages:

• ubuntu-desktop-minimal

• firefox

• virtualbox-guest-additions-iso

• virtualenv

• nmap

• vlan

• ipmitool

• isc-dhcp-server

• tftpd-hpa

• nfs-kernel-server

• nfs-common

• ntp

• curl

• expect

• socat

• ntpdate

• iperf

• yamllint

Next, create the virtualenv. There might already be one already there, but it was packaged on Ubuntu 20.04 (but not a constant upstream), so remove it. Your system python 2.7 might not be the one packaged in the VM installer OVA.

Upload the ISO you want to install ActiveScale from to the VM under `~/<your ISO>`, and run the following snippet:

Do the same as you would in case you started from the standard OVA.

The inventories used in the installer are in the yaml format. This means whitespace determines the structure and precedence of items in the file. Be aware of this! Also mind the difference of TAB and spaces on Unix-based systems of which the installer VM is an example.

Generally speaking, the installer inventory structure looks like this:

So we have 2 types of systems: installers and asnodes. There will always only be one installer defined under installers. In essence this is the installer VM, but specified in the Private VLAN. The number of asnodes is variable. It ranges from 3 for P100E3 to a minimum of 6, all the way up to 36 for X200. Under asnodes we will enter the required info on a per-node basis, allowing for full flexibility.

First step in building the inventory is defining the installer. For this you will need the IP you set on the VM in 3.0 Setup the Quantum Installer VM.

Below is an example;

Next section to tackle is the vars section under asnodes. This will be based upon the customer requirements and networks provided. Great care needs to be taken here, because this models the ActiveScale system fully. For simplicity we will split this up in different subsections.

• deployment_id: Advanced option to override the automatically generated deployment-id by a predefined value

• nr_of_mgmt_nodes:5: If you want 2-node failure resiliency, you can uncomment this, but at the expense of having 50% less metadata capacity as a result. Please pick a storage policy that supports 2-node failure resiliency from 6node_1GEO_2nodeFailure_policies.xlsx

  • 2-node failure resiliency is currently not supported in 3GEO scenarios.

  • This is NOT supported for 3 and 4 node environments. System needs to be 6 nodes or more.

• desired_bmc_user: This is the IPMI/BMC user our rack will use internally. This setting can be set here or on a per node basis. It is important to note this down for future support reference.

• desired_bmc_password: This is the IPMI/BMC password our rack will use internally. This setting can be set here or on a per node basis. It is important to note this down for future support reference.

• datacenter:

  datacenter:

  - name: Site1 # Internal name of datacenter (Will be part of node hostname)

  description: # Description for datacenter

  Here we model the datacenter of the customer. Please note that the name set here will also be part of the ASNode hostname. e.g. Site1-RXX... Also please note the - in front of name. This is needed as for 3GEO we will define multiple datacenters. The description set here will also be visible in the UI.

• racks:

  racks:

  # If mutiple racks are defined here, one must explicitly assign each asnode to a rack by adding a 'location: <rack-name>' statement for each asnode

  - name: R01 # Rack number within datacenter (Will be part of node hostname)

  serial: 'CHANGE_ME' # Serial number of this rack

  Here we define the logical ActiveScale racks. Again the name portion will be part of the hostname. If mutiple racks are defined here, one needs to define the location under this rack. This location needs to match with a datacenter named as defined above, e.g. Site1.

  Don’t forget to adapt the serial here! Set this to the rack serial number as defined on the boxes in which the system was shipped. Our support depends on this serial number to identify a customer’s rack and for linking phone home and alert e-mails.

  Note: Failing to set the serial here correctly, will make the installer fail further on!

• dssconfig: Here we set the DSS storage policy the rack will use. Please refer to the different Excel sheets under ~/quantum_installer and the customer’s requirements to select the appropriate values.

  dssconfig:

  spread_width: 18 # Large file policy spread width, according to choice made by customer. Make sure to pick one of the supported storage policies in the appropriate spreadsheet summary tab. Fill in the template based on the properties of that specific policy.

  blockstore_safety: 5 # Large file policy safety, according to choice made by customer. Make sure to copy the safety value that corresponds to the policy you picked the appropriate spreadsheet

  max_blocks_per_node: 4 # Large file policy data balance rule. Last number to be changed based on 'Max blocks per node' column in the appropriate spreadsheet file, for the specific policy you picked.

• scaleout_rack: Set this to true if this rack is to be used as scale-out rack for an existing ActiveScale system. See the Scale-out guide for more info. Coincidentally, the federated_api section is also only relevant when this rack is a scale-out rack.

• column_number: Defaults to C01, this defines the column number of this rack, should be in format CXX. If this is a scale-out rack should be set to 1 higher than the current highest column numer. E.g for first scale-out rack: C02, for the third scaleout: C03

• model: As mentioned in the template, don’t touch this value as the installer uses it to determine the type of system being installed. Possible values are X200_1GEO for X200, P200_1GEO for P200 and denali_1GEO for P100E3

• networks: The next section defines the private and public networks being used in this install.

  # Defintion of PrivateVLAN1

  # Make sure the startip is the first available IP for ActiveScale if not using explicit IP assignment!

  private1:

  network: 172.20.0.0

  netmask: 255.255.255.0

  startip: 172.20.0.1

  endip: 172.20.0.254

  dhcp_range: 10

  # Defintion of PrivateVLAN2

  # Make sure the startip is the first available IP for ActiveScale if not using explicit IP assignment!

  private2:

  network: 172.20.128.0

  netmask: 255.255.255.0

  startip: 172.20.128.1

  endip: 172.20.128.254

  # Defintion of Customer's Public Network 1

  public1:

  network: 192.168.1.0

  netmask: 255.255.255.0

  startip: 192.168.1.1

  endip: 192.168.1.254

  routes:

  # Below is the default route for the public networks

  - name: PublicNetwork1_route

  destination_network: 0.0.0.0

  destination_netmask: 0.0.0.0

  destination_gateway: 192.168.1.1 # Default gateway

  # Defintion of Customer's Public Network 2

  public2:

  network: 192.168.2.0

  netmask: 255.255.255.0

  startip: 192.168.2.1

  endip: 192.168.2.254

  Note: While the default for the private VLANs should suffice, it could be a customer has different requirements. Make sure to collect the correct values for the public networks being used. Setting these here correctly prevents a lot of work once the system is installed, because the public interfaces will be reachable immediately.

  The above definitions are quite self-explanatory, it’s important to note that if the different private IPs are not set explicitly on a per node basis, the installer will use the value of startip as the first IP to assign to a node in a logical order. Also, if NIC bonding is used (see below), you do not need to define public2 because it’s not used. We recommend assigning the entire subnet (startip and endip), although if you must limit it, note that there need to be enough IPs for each machine and the IPMI for each machine in private1, which means at least 2 times (number of machines)!

  Note: Please note when defining public1_ip_address and/or public2_ip_address on ASNode level (see below), public1 and/or public2 defined here need to match. So don’t leave the defaults for these networks if the customer is going to use anything else than these defaults.

  Next you can define if NIC bonding on the public NICs will be used or not. If this is not the case you can safely ignore this section and leave bonded:false in place. Otherwise put it to true and configure the other options as needed. Example:

  bonded: true

  bonding:

  # Bonding mode

  # Choices are roundrobin, lacp or alb

  mode: lacp

  # Link monitoring frequency in milliseconds (default 100)

  miimon: 100

  # The rate in which the driver requests its link partner to transmit LACPDU packets in lacp mode

  # Choices are fast or slow

  lacp_rate: fast

  # The transmit hash policy to use for slave selection, applicable to modes lacp and alb

  # Choices are layer2, layer2+3 or layer3+4

  xmit_hash_policy: layer3+4

The final section to define in the inventory determines the amount of nodes, and which physical node is which. For simplicity we have defined 3 nodes for P100E3 and 6 for X200 and P200. To expand the number of nodes for X200, you simply need to add the next CMB0X definition and define all parameters. If you’re installing a P200 system with less than 6 nodes, remove the unsued defintions!

For each node there are 2 required parameters and 4 optional.

If the BMC/IPMI is fully configured when the rack is installed and a (default) username and password are known, we can opt to use the fully automated install flow. Via this flow it is no longer needed for an operator to manually power-on/cycle the ASnodes when PXE-booting. To achieve this we need to edit one general config file and add some extra data in the inventory.

4.6.1 Configure IPMI Address

Configure the IPMI IP address from the BIOS setup utility for each node by performing the following:

1. On boot-up when the Quantum screen is displayed, select the DEL key until the system enters the BIOS setup utility.

   

   The setup utility will open at the Main menu

   

2. Tab or arrow over to the IPMI tab and select Enter.

   

3. Select BMC Network Configuration and press Enter.

   The BMC network configuration will display.

   

4. Select Update IPMI Lan configuration [Yes].

5. Scroll down to Configuration Address Source and select Static.

   

6. Scroll down to Station IP Address and select Enter.

   

7. Enter the IPMI IP Address of the node and select Enter.

8. Scroll down to Subnet Mask and select Enter.

   

9. Enter the Subnet Mask for the node and select Enter.

10. Select Escape (ESC).

11. Tab over to Save & Exit and select Enter.

    

12. Select Save Changes and Reset and select Enter.

13. Select Yes. The Quantum screen will display.

    

14. Repeat Steps 1 - 13 for each node in the P200 system.

4.6.2 Configure the IPMI Username and Password

1. Enter the IPMI IP Address into the browser from within the Quantum VM Installer. The login prompt will display.

   

2. Enter the username ADMIN and the password.

   X200

   Password is on the label on the twin server.

   

   Note: Server number one is the bottom server, server two is the top.

   P200

   Password is on the back of the service tag on the server.

   

3. Select the Configuration tab and chooser Users.

   

4. Select an empty User ID line, it will highlight in yellow.

   

5. Select line 10 and enter marvin as the user name.

   

6. Enter the marvin user name and enter the password Act1v3Sc4l3!#.

   

7. The new user added screen will display.

   

8. Select Logout.

9. Verify the added user by entering the Username marvin and the Password Act1v3Sc4l3!#.

   

4.6.3 Verify BIOS Settings

Verify BIOS for IPV4 (only) PXE Boot by performing the following:

1. Upon boot-up, select del at the Quantum screen until the BIOS utility opens.

   

   The BIOS utility will open at the main menu

   

2. Select the Boot tab and select Enter.

   

3. Scroll down to UEFI NETWORK Drive BBS Priorities and select Enter.

   

4. Verify Options 1through 4 are PXE IPV4 only and that 5 through 8 are disabled. There should be no IPV6 options.

This step checks if the installer can reach itself over the Private VLAN (basic sanity check) and also extracts the ISO and configure TFTP. We opted to do this here, to shrink the size of the OVA as much as possible.

To run this step, either open a terminal on the VM or ssh into the VM from the host via: sshinstaller@127.0.0.1-p2222 and execute the following:

Next, execute the following:

Where <inventory> should be the name of the inventory file that was edited in 4 Building the inventory, but without the .yaml extension.

5.2.1 Usage

This step replaces the tedious USB boot part from the field installation procedure by PXE booting the ASnodes and installing our ActiveScale to disk automatically based upon the data entered in the inventory. This step also allows for easier re-do of the install if something goes wrong in the final step.

This step consists of the installer setting up al needed DHCP rules, the operator manually powering on/cycle all ASnodes and then the installer assigning an IP-address and performing the PXE-boot and install.

Note: The manual power-on/cycle by the operator can be circumvented by having BMC/IPMI configured at rack time. See: 4.6 Opting for the Fully Automated Install (Recommended)

1. Setup DHCP rules

2. Installer prompt operator to power cycle/on the ASnodes

3. ASnodes send DHCPDISCOVER and get an IP address from the VM installer

4. VM installer serves a PXE boot image to the ASnodes

5. ASnodes boot from this PXE image and request the same IP once more

6. Installer waits until all ASnodes are up and reachable over SSH or it times out and notifies the operator

7. Once all ASnodes are reachable over SSH, they are NTP time synced with the VM installer

8. The ActiveScale software is deployed to disk and the node info (IP,name, ..) are persisted to disk

9. ASnodes are rebooted from disk

10. Installer checks if all ASnodes are booted from disk

Note: We recommend to power off all ASnodes before starting the installer so that all nodes are roughly in the same state at the same time. It will also ease potential debugging because if something is wrong, a certain node will run behind in the output. Leave the JBODs powered on!

To run this step, either open a terminal on the VM or ssh into the VM from the host via: sshinstaller@127.0.0.1-p2222 and execute the following:

Where <inventory> should be the name of the inventory file that was edited in 4 Building the inventory, but without the .yaml extension.

When not opting for the fully automatic approach, after a short while, the output will pause at:

This is the point where you need to manually power on the ASNodes.

From the moment where the nodes are powered on/cycled until the next output it should take around 10/15 minutes. You can always follow the PXE boot process live on ASnode by connecting a display to one of the nodes.

5.2.2 Basic Debug

The most common issues an operator can encounter here is either a mistake in the inventory or a hardware issue.

MAC Address

It’s critically important to have the MAC-address correct in the inventory before running this step. If you notice that a node is not PXE-booted or much slower then the others to PXE-boot, it is best to check which MAC-address is being seen on the switch on the corresponding port. If this matches with the input in the inventory you can check the syslog of the installer VM to see which MAC-addresses are sending a DHCPDISCOVER.

DHCP Server

Also make sure that no other DHCP servers are running on the Private VLAN. This is especially the case with customer’s using shared switches instead of dedicated switches. Having an other DHCP server present in this VLAN will cause collisions on the network.

P100E3

For P100E3 specific, please make sure the correct NIC is set as first boot NIC in the BIOS. Should be: NIC2 IPv4 Intel(R) Ethernet Connection X552 10 GbE Backplane.

Bandwidth

If the installation fails during PXE-boot with the message there’s not enough bandwidth available, please double check how much bandwidth there is available between the VM installer and the ActiveScale system. By default we require and check on a value around 1Gbit/sec (actually 0,9 Gbit/sec as a practical minimum). If this value is not present, and you want to overwrite this maximum, you can do this by adding the following to you inventory file:

Where 700000000 is the amount of bits per second you want as a minimum. Please also note that how lower the available bandwidth, the longer it will take to PXE-boot and perform the install to disk, because we need to copy around 3GB of data per ActiveScale node during this install. It can very well be you run into timeouts. These are set to 1500 seconds by default. If you want to overwrite these, you can do so by editing: inventories/group_vars/all.yaml

Add a definition for himalaya_install_timeout and set a new value, e.g. 2000. To extend the timeout when deploying ActiveScale to disk, the default value is 1500s. In the case the install fails during the post-install phase you can define himalaya_post_install_timeout to override this timeout. Default value is 600s (10 minutes).

Please take into account that the post-install consists of copying about 3GB of data from the installer to each node, so in low bandwidth situations this can cause havoc. In very rare cases you need to define a new value for pxe_timeout. This is the time the installer maximum waits before a node should be PXE-booted ready to receive further instructions. Most pxe_timeout related errors are caused by cabling errors or incorrect BIOS boot settings. Or: the operator not being quick enough to power cycle the nodes in the manual power cycle scenario.

This step configures the ASnodes fully as ActiveScale systems, by installing all the different components, configuring them, formatting disks, assigning and activating blockstores, apply policies, …

At the end of this step, the system will be ready to run the mini CBW to set the final settings.

All queue files which needed to be altered manually in the field installation procedure are generated automatically here based upon the data entered in the inventory. This step consists of the following subtasks:

1. Generate the different queue files and config files

2. Copy these to CMB01

3. Run the install_activescale.yaml playbook on CMB01

This step will take several hours to complete and total run time is dependent on the amount of disks and total config of the ASnodes. We estimate a minimal run time of 4 to 6 hours.

To run this step, either open a terminal on the VM or ssh into the VM from the host via: sshinstaller@127.0.0.1-p2222 and execute the following:

Where <inventory> should be the name of the inventory file that was edited in 4 Building the inventory, but without the .yaml extension.

Situations can arise where one needs to connect to one of the nodes from the installer node without the use of a username/password. This can be done by using the factory installer ssh key. Please note that after CBW is installed, the default factory ssh key is replaced.

The key (with name id_rsa_factory) can be found in the quantum_installer/keys folder. One can also locate the factory installer ssh key by using:

The found key can be used to connect to a node:

where KEY denotes the key’s path and NODE_IP the IP address of the node one wants to connect to.

An ActiveScale 3GEO system, is an ActiveScale system spread accross 3 different physical sites. This means there are 3 times the amount of network subnets involved. It’s up to the customer to decide how he sets up the public networking side of things, but for ActiveScale, each ActiveScale instance in each site has 2 seperate private networks which need to be able to access one of the private networks in the other sites to spread the data.

Most customers opt to use a WAN connection to transparently connect all private vlans and we simply have to add a route in ActiveScale to a specific gateway which is part of the local private network. In this guide we will not explain the 3GEO networking setup, but having a key understanding on how this all ties together is paramount in understanding how to correctly use the VM installer.

The VM installer PXE boots all the different nodes of an ActiveScale system to be able to install them. With 3GEO however, it’s possible that there is simply not enough bandwidth to PXE boot the nodes in Site 2 or 3 from a VM Installer which is located in Site 1. It could be that the PXE boot step needs to be performed seperately in 3 distinct physical locations, this guide will explain which commands to use to achieve this. It’s paramount however that the inventory used in all 3 sites matches and to achieve this, we recommend building the inventory before heading out to the customer’s sites to do the actual install. It’s much easier to spot potential issues in the inventory when not being stressed about performing the actual install.

Note: It’s important that the routing between the different private networks in the different sites is up and running when doing the install. Especially because the actual deployment of ActiveScale will happen from Site 1.

Note: You will notice that Site 1 in a 3GEO setup can be seen as the ‘parent’ site. Next to the actual deployment of ActiveScale being started here, it’s important to note that also the mini-CBW can only be run from Site 1! It will configure all sites, but needs to be started from Site 1.

The inventory for a 3GEO system is in essence a folder in the inventories subfolder of the VM Installer. It consists of some additional configuration files and 3 inventory files. Each of these 3 inventory files are very similar to the inventory file of a single GEO system, as they also model 1 specific GEO.

The 3 files are named: GEO1.yaml, GEO2.yaml and GEO3.yaml. There are 2 very important differences between the different files: - Only GEO1.yaml can have the stroage policy configuration present because this configuration needs to be consistent accross all GEOs - The definition of the private networks WILL differ in each GEO, the definition of the public networks CAN differ. Either way both private and public networks need to be defined and modeled correctly for each GEO even if they are the same.

In the case where we cannot PXE-boot accross different GEOs, also the definition in the installers section of each file will differ, even if you re-use the same VM with the same set of files, because of the fact the private networks differ, the IP of the installer in each GEO will differ. Don’t forget to change the installer IP in each GEO accordingly! We recommend to use a similar IP in each GEO, keep the last octet the same and simply alter the subnet portion of the IP according to the definition of each private network in the GEOs.

To ease the configuration, we have provided a folder and example files (templates) for each supported hardware type: X200 and P200. These can be found under ~/quantum_installer/inventories/3GEO-X200 and ~/quantum_installer/inventories/3GEO-P200. The only files which should be altered without the support of engineering are GEO1.yaml, GEO2.yaml and GEO3.yaml.

Note: Don’t forget to edit all 3 files! Especially things like serial number, etc are key differences across GEOs

The templates have some explanation internally, but for a full overview you can have a look at: 4.0 Build the Inventory. Below we will explain some key differences:

As mentioned above the main difference in the inventory is the fact that for 3GEO the inventory is actually a folder of files instead of 1 single file. Depending on the scenario defined below, you will have to point the script to either the folder or a specific file.